Papers with embedding layers
Accelerating LLM Fine-Tuning via Embedding Knowledge Transfer (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on parameter-efficient fine-tuning (PEFT) have produced many state-of-the-art results by adapting LLMs to new tasks, but it requires substantial training data and time to enhance model performance. |
| Approach: | They propose a parameter-efficient fine-tuning framework which efficiently transfers knowledge from a small expert model to a target large model via embedding layers. |
| Outcome: | The proposed framework accelerates domain-specific fine-tuning, improves model performance and remains robust across diverse model families and PEFT methods. |
Investigating grammatical abstraction in language models using few-shot learning of novel noun gender (2024.findings-eacl)
Copied to clipboard
| Challenge: | a new study shows that language models can generalise novel noun gender from one to two learning examples and apply it across agreement contexts. |
| Approach: | They conduct a noun learning experiment to assess whether a transformer and an LSTM can achieve human-like abstraction of grammatical gender in French. |
| Outcome: | The proposed models generalise gender from one to two learning examples and apply gender across agreement contexts, albeit with a bias for the masculine gender category. |
Vocabulary Adaptation for Domain Adaptation in Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Neural network methods exhibit strong performance only in a few resource-rich domains. |
| Approach: | They propose a method that fine-tunes embedding layers of a pre-trained NMT model to the target domain. |
| Outcome: | The proposed method improves fine-tuning performance in En-Ja and De-En translation by 3.86 and 3.28 BLEU points. |
MulCode: A Multiplicative Multi-way Model for Compressing Neural Language Model (D19-1)
Copied to clipboard
| Challenge: | a large number of parameters dominate the memory usage of deep neural nets . compression of embedding layers is the key to reducing memory usage . |
| Approach: | They propose a multi-way multiplicative neural compressor to compress embedding layers . they use an adaptively created matrix and multiplicativative compositions to learn them . |
| Outcome: | a new multi-way multiplicative neural compressor can achieve 41.38 times compression rate with little loss in performance. |
Language Models Can be Efficiently Steered via Minimal Embedding Layer Transformations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) neglect the embedding layer. |
| Approach: | They propose a PEFT approach that modifies input embeddings without altering hidden layers. |
| Outcome: | Experiments show that TinyTE modifies embeddings without altering hidden layers . the proposed approach achieves competitive performance while requiring 0.0001% of parameters . |
Quantized Can Still Be Calibrated: A Unified Framework to Calibration in Quantized Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to quantify uncertainty of large language models (LLMs) but their influence on uncertainty calibration remains unexplored. |
| Approach: | They propose an analytic method to estimate the upper bound of calibration error (UBCE) for quantized LLMs and propose a method to recover calibration errors through soft-prompt tuning. |
| Outcome: | The proposed method improves the calibration accuracy of quantized models on multiple datasets and LLMs. |